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Abstract 

Customer care in technical domains is increasingly 
based on e-mail communication, allowing for the re- 
production of approved solutions. Identifying the 
customer's problem is often time-consuming, as the 
problem space changes if new products are launched. 
This paper describes a new approach to the classifi- 
cation of e-mail requests based on shallow text pro- 
cessing and machine learning techniques. It is im- 
plemented within an assistance system for call center 
agents that is used in a commercial setting. 

1 Introduction 

Customer care in technical domains is increasingly 
based on e-mail communication, allowing for the re- 
production of approved solutions. For a call cen- 
ter agent, identifying the customer's problem is of- 
ten time-consuming, as the problem space changes 
if new products are launched or existing regulations 
are modified. The typical task of a call center agent 
processing e-mail requests consists of the following 
steps: 

Recognize the problem(s): read and understand 
the e-mail request; 

Search a solution: identify and select predefined 
text blocks; 

Provide the solution: if necessary, customize 
text blocks to meet the current request, and 
send the text. 

This task can partly be automated by a system 
suggesting relevant solutions for an incoming e-mail. 
This would cover the first two steps. The last step 
can be delicate, as its primary goal is to keep the 
customer satisfied. Thus human intervention seems 
mandatory to allow for individual, customized an- 
swers. Such a system will 

• reduce the training effort required since agents 
don't have to know every possible solution for 
every possible problem; 

• increase the agents' performance since agents 
can more quickly select a solution among several 
offered than searching one; 



• improve the quality of responses since agents 
will behave more homogeneously - both as a 
group and over time - and commit fewer errors. 

Given that free text about arbitrary topics must 
be processed, in-depth approaches to language un- 
derstanding are not feasible. Given further that the 
topics may change over time, a top-down approach 
to knowledge modeling is out of the question. Rather 
a combination of shallow text processing (STP) with 
statistics-based machine learning techniques (SML) 
is called for. STP gathers partial information about 
text such as part of speech, word stems, negations, 
or sentence type. These types of information can be 
used to identify the linguistic properties of a large 
training set of categorized e-mails. SML techniques 
are used to build a classifier that is used for new, 
incoming messages. Obviously, the change of topics 
can be accommodated by adding new categories and 
e-mails and producing a new classifier on the basis 
of old and new data. We call this replacement of a 
classifier "relearning" . 

This paper describes a new approach to the clas- 
sification of e-mail requests along these lines. It is 
implemented within the ICC-mail system, which 
is an assistance system for call center agents that 
is currently used in a commercial setting. Section || 
describes important properties of the input data, i.e. 
the e-mail texts on the one hand, and the categories 
on the other. These properties influenced the system 
architecture, which is presented in Section ||. Vari- 
ous publicly available SML systems have been tested 
with different methods of STP-based preprocessing. 
Section || describes the results. The implementation 
and usage of the system including the graphical user 
interface is presented in Section ||. We conclude by 
giving an outlook to further expected improvements 
(Section ^|). 

2 Data Characteristics 

A closer look at the data the ICC-mail system is 
processing will clarify the task further. We carried 
out experiments with unmodified e-mail data accu- 
mulated over a period of three months in the call 
center database. The total amount was 4777 e-mails. 



We used 47 categories, which contained at least 30 
documents. This minimum amount of documents 
turned out to render the category sufficiently dis- 
tinguishable for the SML tools. The database con- 
tained 74 categories with at least 10 documents, but 
the selected ones covered 94% of all e-mails, i.e. 4490 
documents. 

It has not yet generally been investigated h ow the 
type of data influences the learning result ( Yang J 
1999] ), or under which circumstances which kind of 



preprocessing and which learning algorithm is most 
appropriate. Several aspects must be considered: 
Length of the documents, morphological and syn- 
tactic well-formedness, the degree to which a docu- 
ment can be uniquely classified, and, of course, the 
language of the documents. 

In our application domain the documents differ 
very much from documents generally used in bench- 
mark tests, for example the Reuters corpus]]. First 
of all, we have to deal with German, whereas the 
Reuters data are in English. The average length of 
our e-mails is 60 words, whereas for documents of 



Rcutcrs-21578 it is 129 words. The number of cat- 
egories we used compares to the top 47 categories 
of the Reuters TOPICS category set. While we 
have 5008 documents, TOPICS consists of 13321 in- 
stance^. The Reuters documents usually are mor- 
phologically and syntactically well-formed. As e- 
mails are a more spontaneously created and infor- 
mal type of document, they require us to cope with 
a large amount of jargon, misspellings and gram- 
matical inaccuracy. A drastic example is shown in 
Figure |^. The bad conformance to linguistic stan- 
dards was a major argument in favor of STP instead 
of in-depth syntactic and semantic analysis. 

The degree to which a document can be uniquely 
classified is hard to verify and can only be inferred 
from the results in general terms .0 It is, however, 
dependent on the ability to uniquely distinguish the 
classes. In our application we encounter overlapping 
and non-exhaustive categories as the category sys- 
tem develops over time. 

3 Integrating Language Technology 
With Machine Learning 

STP and SML correspond to two different 
paradigms. STP tools used for classification tasks 
promise very high recall/precision or accuracy val- 
ues. Usually human experts define one or several 
template structures to be filled automatically by ex- 
t racting information fr om the documents (cf. e.g. 
( Ciravegna et al., 1999] )). Afterwards, the partially 



^ttp : //vw. research. att . com/~lewis/reuters2157s| . 
html 

2 We took only uniquely classified documents into account 

3 Documents containing multiple requests can at present 
only be treated manually, as described in Section 



filled templates are classified by hand-made rules. 
The whole process brings about high costs in analyz- 
ing and modeling the application domain, especially 
if it is to take into account the problem of changing 
categories in the present application. 

SML promises low costs both in analyzing and 
modeling the application at the expense of a lower 
accuracy. It is independent of the domain on the 
one hand, but does not consider any domain specific 
knowledge on the other. 

By combining both methodologies in ICC-MAIL, 
we achieve high accuracy and can still preserve a use- 
ful degree of domain-independence. STP may use 
both general linguistic knowledge and linguistic al- 
gorithms or heuristics adapted to the application in 
order to extract information from texts that is rele- 
vant for classification. The input to the SML tool is 
enriched with that information. The tool builds one 
or several categorizers^ that will classify new texts. 

In general, SML tools work with a vector represen- 
tation of data. First, a relevancy vector of relevant 



features for each class is computed (Yang and Ped 



ersen, 1997). In our case the relevant features con- 



sist of the user-defined output of the linguistic pre- 
processor. Then each single document is translated 
into a vector of numbers isomorphic to the defining 
vector. Each entry represents the occurrence of the 
corresponding feature. More details will be given in 
Section § 

The ICC-mail architecture is shown in Figure [l]. 
The workflow of the system consists of a learning 
step carried out off-line (the light gray box) and an 
online categorization step (the dark gray box). In 
the off-line part, categorizers are built by processing 
classified data first by an STP and then by an SML 
tool. In this way, categorizers can be replaced by the 
system administrator as she wants to include new 
or remove expired categories. The categorizers are 
used on-line in order to classify new documents after 
they have passed the linguistic preprocessing. The 
resulting category is in our application associated 
with a standard text that the call center agent uses 
in her answer. The on-line step provides new clas- 
sified data that is stored in a dedicated ICC-mail 
database (not shown in Figure [l]). The relearning 
step is based on data from this database. 

3.1 Shallow Text Processing 

Linguistic preprocessing of text documents is car- 
ried out by re-using smes, an information extrac- 
tion cor e system for real-world German text pro- 
cessing ( Neumann et al., 1997 ). The fundamental 
design criterion of smes is to provide a set of basic, 
powerful, robust, and efficient STP components and 



4 Almost all tools we examined build a single multi- 
categorizer except for SVM_Light, which builds multiple bi- 
nary classifiers. 
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Figure 1: Architecture of the ICC-MAIL System. 



generic linguistic knowledge sources that can eas- 
ily be customized to deal with different tasks in a 
flexible manner, smes includes a text tokenizer, a 
lexical processor and a chunk parser. The chunk 
parser itself is subdivided into three components. In 
the first step, phrasal fragments like general nominal 
expressions and verb groups are recognized. Next, 
the dependency-based structure of the fragments of 
each sentence is computed using a set of specific sen- 
tence patterns. Third, the grammatical functions 
are determined for each dependency-based structure 



on the basis of a large subcategonzation lexicon. 



the present application benchts trom the high mod- 
ularity of the usage of the components. Thus, it is 
possible to run only a subset of the components and 
to tailor their output. The experiments described in 
Section |] make use of this feature. 

3.2 Statistics-Based Machine Learning 

Several SML tools representing different learning 
paradigms have been selected and evaluated in dif- 
ferent settings of our domain: 

Lazy Learning: Lazy Learners are also known 
as memory-based, instance-based, exemplar- 
based, case-based, experience-based, or k- 
nearest neighbor algorithms. They store all 
documents as vectors during the learning phase. 
In the categorization phase, the new document 
vector is compared to the stored ones and is 
categorized to same class as the fc-nearest neigh- 
bors. The distance is measured by computing 
e.g. the Euclidean distance between the vectors. 
By changing the number of neighbors k or the 
kind of distance measure, the amount of gener- 
alization can be controlled. 
We used IB ( |Aha, 1992]), which is part of 
the MLC++ library (Kohavi and Sommcrficld 



1996) 



Symbolic Eager Learning: This type of learners 
constructs a representation for document vec- 
tors belonging to a certain class during the 
learning phase, e.g. decision trees, decision rules 
or probability weightings. During the catego- 
rization phase, the representation is used to as- 
sign the appropriate class to a new document 
vector. Several pruning or specialization heuris- 
tics can be used to control the amount of gen- 
eralization. 



We u sed ID3 ( |Quinlan, 1986]), C4.5 (|Quinlan 



1992] ) and C5.0, RIPPER. ( Johcn, 199 
the Naive Bayes inducer (Glood, 196 



and 
con- 
tained in the MLC++ library. ID3, C4.5 and 
C5.0 produce decision trees, RIPPER is a rule- 
based learner and the Naive Bayes algorithm 
computes conditional probabilities of the classes 
from the instances. 

Support Vector Machines (S VMs): SVMs are 
described in ( Vapnik, 1995 ). SVMs are binary 
learners in that they distinguish positive and 
negative examples for each class. Like eager 
learners, they construct a representation dur- 
ing the learning phase, namely a hyper plane 
supported by vectors of positive and negative 
examples. For each class, a categorizer is built 
by computing such a hyper plane. During the 
categorization phase, each categorizer is applied 
to the new document vector, yielding the prob- 
abilities of the document belonging to a class. 
The probability increases with the distance of 
thevector from the hyper plane. A document 
is said to belong to the class with the highest 
probability. 



We chose SVMXight (Joachims, 1998) 



Neural Networks: Neural Networks are a special 
kind of "non-symbolic" eager learning algo- 



rithm. The neural network links the vector el- 
ements to the document categories The learn- 
ing phase defines thresholds for the activation 
of neurons. In the categorization phase, a new 
document vector leads to the activation of a sin- 



gle categ ory. For details we refer to ( Wiener ct 
al., 1995|). 



In our application, we tried out the Learning 



Vecto r Quantization (LVQ) ( Kohoncn et al. J 
1996 ). LVQ has been used in its default config- 



uration only. No adaptation to the application 
domain has been made. 

4 Experiments and Results 

We describe the experiments and results we achieved 
with different linguistic preprocessing and learning 
algorithms and provide some interpretations. 

We start out from the corpus of categorized e- 
mails described in Section ^. In order to normalize 
the vectors representing the preprocessing results of 
texts of different le ngth, and to concentrate o n rel- 
evant material (cf. ( Yang and Pedersen, 1997[ )), we 
define the relevancy vector as follows. First, all doc- 
uments are preprocessed, yielding a list of results 
for each category. From each of these lists, the 100 
most frequent results - according to a TF/IDF mea- 
sure - are selected. The relevancy vector consists of 
all selected results, where doubles are eliminated. 
Its length was about 2500 for the 47 categories; it 
slightly varied with the kind of preprocessing used. 

During the learning phase, each document is pre- 
processed. The result is mapped onto a vector of 
the same length as the relevancy vector. For ev- 
ery position in the relevancy vector, it is determined 
whether the corresponding result has been found. In 
that case, the value of the result vector element is 1, 
otherwise it is 0. 

In the categorization phase, the new document is 
preprocessed, and a result vector is built as described 
above and handed over to the categorizer (cf. Fig- 
ure|). 

While we tried various kinds of linguistic prepro- 
cessing, systematic experiments have been carried 
out with morphological analysis (MorphAna), shal- 
low parsing heuristics (S TP- Heuristics), and a com- 
bination of both (Combined) . 

MorphAna: Morphological Analysis provided by 
smes yields the word stems of nouns, verbs and 
adjectives, as well as the full forms of unknown 
words. We are using a le xicon of approx. 10000 
word stems of German ( Neumann et al., 1997 ) . 

STP-Heuristics: Shallow parsing techniques are 
used to heuristically identify sentences contain- 
ing relevant information. The e-mails usually 
contain questions and/or descriptions of prob- 
lems. The manual analysis of a sample of 



the data suggested some linguistic constructions 
frequently used to express the problem. We ex- 
pected that content words in these construc- 
tions should be particularly influential to the 
categorization. Words in these constructions 
are extracted and processed as in MorphAna, 
and all other words are ignored.^] The heuris- 
tics were implemented in ICC-mail using smes. 

The constructions of interest include negations 
at the sentence and the phrasal level, yes-no 
and wh-questions, and declaratives immediately 
preceding questions. Negations were found to 
describe a state to be changed or to refer to 
missing objects, as in / cannot read my email 
or There is no correct date. We identified them 
through negation particles |] Questions most of- 
ten refer to the problem in hand, either directly, 
e.g. How can I start my email program? or in- 
directly, e.g. Why is this the case?. The lat- 
ter most likely refers to the preceding sentence, 
e.g. My system drops my e-mails. Questions are 
identified by their word order, i.e. yes-no ques- 
tions start with a verb and wh-questions with a 
wh-particle. 

Combined: In order to emphasize words found 
relevant by the STP heuristics without losing 
other information retrieved by MorphAna, the 
previous two techniques are combined. Empha- 
sis is represented here by doubling the number 
of occurrences of the tokens in the normaliza- 
tion phase, thus increasing their TF/IDF value. 

Call center agents judge the performance of ICC- 
mail most easily in terms of accuracy: In what per- 
centage of cases does the classifier suggest the correct 
text block? In Table 0, detailed information about 
the accuracy achieved is presented. All experiments 
were carried out using 10-fold cross-validation on the 
data described in Section [|. 

In all experiments the SVM_Light system outper- 
formed other learning algor ithms, which confirms 
Yang's ( |Yang and Liu, 1999| ) results for SVMs fed 
with Reuters data. The fc-nearest neighbor algo- 
rithm IB performed surprisingly badly although dif- 
ferent values of k were used. For IB, ID3, C4.5, C5.0, 
Naive Bayes, RIPPER and SVM_Light, linguis- 
tic preprocessing increased the overall performance. 
In fact, the method performing best, SVM_Light, 
gained 3.5% by including the task-oriented heuris- 
tics. However, the boosted RIPPER and LVQ scored 
a decreased accuracy value there. For LVQ the de- 
crease may be due to the fact that no adaptations to 



5 If no results were found this way, MorphAna was applied 
instead. 

6 We certainly would have benefited from lexical semantic 
information, e.g. The correct date is missing would not be 
captured by our approach. 





SML algorithm 


MorphAna 

Best Best5 


STP-Heuristics 

Best Best5 


Combined 

Best Best5 


Neural Nets 


LVQ 


35.66 


22.29 


25.97 


Lazy Learner 


IB 


33.81 


33.01 


35.14 


Symbolic Eager 


Naive Bayes 


33.83 


33.76 


34.01 


Learners 


ID3 


38.53 


38.11 


40.02 




RIPPER 


47.08 


49.38 


50.54 




Boosted Ripper 


52.73 


49.96 


50.78 




C4.5 


52.00 


52.90 


53.40 




C5.0 


52.60 


53.20 


54.20 


Support Vectors 


SVM_Light 


53.85 74.91 


54.84 78.05 


56.23 78.17 



Table 1: Results of Experiments. Most SML tools deliver the best result only. SVM_Light produces ranked 
results, allowing to measure the accuracy of the top five alternatives (Best5). 



the domain were made, such as adapting the number 
of codebook vectors, the initial learning param eters 



or the number of i terations during training (cf. (Ko- 



honcn ct al., 1996[ )). Neural networks are rather sen- 
sitive to misconfigurations. The boosting for RIP- 
PER seems to run into problems of overfitting. We 
noted that in six trials the accuracy could be im- 
proved in Combined compared to MorphAna, but in 
four trials, boosting l ed to deteriora tion. This effect 
is also mentioned in (Quinlan, 1996). 

These fi gures are slightly lower than th e ones re- 
ported by ( Neumann and Schmcicr, 1999] ) that were 
obtained from a different data set. Moreover, these 
data did not contain multiple queries in one e-mail. 

It would be desirable to provide explanations for 
the behavior of the SML algorithms on our data. As 
we have emphasized in Section ^, general methods 
of explanation do not exist yet. In the application 
in hand, we found it difficult to account for the ef- 
fects of e.g. ungrammatical text or redundant cate- 
gories. For the time being, we can only offer some 
speculative and inconclusive assumptions: Some of 
the tools performing badly - IB, ID3, and the Naive 
Bayes inducer of the MLC++ library - have no or 
little pruning ability. With rarely occurring data, 
this leads to very low generalization rates, which 
again is a problem of overfitting. This suggests that 
a more canonical representation for the many ways 
of expressing a technical problem should be sought 
for. Would more extensive linguistic preprocessing 
help? 

Other tests not reported in Table [j] looked at im- 
provements through more general and sophisticated 
STP such as chunk parsing. The results were very 
discouraging, leading to a significant decrease com- 
pared to MorphAna. We explain this with the bad 
compliance of e-mail texts to grammatical standards 
(cf . the example in Figure ||) . 

However, the practical usefulness of chunk parsing 
or even deeper language understanding such as se- 



mantic analysis may be questioned in general: In a 
moving domain, the coverage of linguistic knowledge 
will always be incomplete, as it would be too expen- 
sive for a call center to have language technology 
experts keep pace with the occurrence of new top- 
ics. Thus the preprocessing results will often differ 
for e-mails expressing the same problem and hence 
not be useful for SML. 

As a result of the tests in our application domain, 
we identified a favorite statistical tool and found that 
task-specific linguistic preprocessing is encouraging, 
while general STP is not. 

5 Implementation and Use 

In this section we describe the integration of the 
ICC-mail system into the workflow of the call cen- 
ter of AOL Bertelsmann Online GmbH & Co. KG, 
which answers requests about the German version 
of AOL software. A client/server solution was built 
that allows the call center agents to connect as 
clients to the ICC-MAIL server, which implements 
the system described in Section ^. For this purpose, 
it was necessary to 

• connect the server module to AOL's own Sybase 
database that delivers the incoming mail and 
dispatches the outgoing answers, and to ICC- 
mail 's own database that stores the classified 
e-mail texts; 

• design the GUI of the client module in a self- 
explanatory and easy to use way (cf . Figure ||) . 

The agent reads in an e-mail and starts ICC-mail 
using GUI buttons. She verifies the correctness of 
the suggested answer, displaying and perhaps se- 
lecting alternative solutions. If the agent finds the 
appropriate answer within these proposals, the asso- 
ciated text is filled in at the correct position of the 
answer e-mail. If, on the other hand, no proposed 
solution is found to be adequate, the ICC-MAIL tool 
can still be used to manually select any text block 
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Figure 2: The GUI of the ICC-MAIL Client. All labels and texts were translated by the authors. The English 
input is based on the following original text, which is similarly awkward though understandable: Wie mache 
ich zum mein Programm total deinstalieren, und wieder neu instalierem, mil, wen Sie mir senden Version 
4.0 ??????????????. The suggested answer text is associated with the category named "Delete & Reinstall 
AOL 4.0" . Four alternative answers can be selected using the tabs. The left-hand side window displays the 
active category in context. 



from the database. The ICC-mail client had to pro- 
vide the functionality of the tool already in use since 
an additional tool was not acceptable to the agents, 
who are working under time pressure. 

In the answer e-mail window, the original e-mail 
is automatically added as a quote. If an e-mail con- 
tains several questions, the classification process can 
be repeated by marking each question and iteratively 
applying the process to the marked part. The agent 
can edit the suggested texts before sending them off. 
In each case, the classified text together with the se- 
lected category is stored in the ICC-mail database 
for use in future learning steps. 

Other features of the ICC-mail client module in- 
clude a spell checker and a history view. The latter 
displays not only the previous e-mails of the same 
author but also the solutions that have been pro- 
posed and the elapsed time before an answer was 
sent. 

The assumed average time for an agent to an- 
swer an e-mail is a bit more than two minutes with 
AOL's own mail processing system.]] With the ICC- 
mail system the complete cycle of fetching the mail, 
checking the proposed solutions, choosing the ap- 
propriate solutions, inserting additional text frag- 
ments and sending the answer back can probably 
be achieved in half the time. Systematic tests sup- 

7 This system does not include automatic analysis of mails. 



porting this claim are not completed yetjj but the 
following preliminary results are encouraging: 

• A test under real-time conditions at the call- 
center envisaged the use of the ICC-MAIL sys- 
tem as a mail tool only, i.e. without taking ad- 
vantage of the system's intelligence. It showed 
that the surface and the look-and-feel is ac- 
cepted and the functionality corresponds to the 
real-time needs of the call center agents, as users 
were slightly faster than within their usual en- 
vironment. 

• A preliminary test of the throughput achieved 
by using the STP and SML technology in ICC- 
mail showed that experienced users take about 
50-70 seconds on average for one cycle, as de- 
scribed above. This figure was gained through 
experiments with three users over a duration of 
about one hour each. 

Using the system with a constant set of categories 
will improve its accuracy after repeating the off-line 
learning step. If a new category is introduced, the 
accuracy will slightly decline until 30 documents are 
manually classified and the category is automatically 
included into a new classifier. Relearning may take 
place at regular intervals. The definition of new cat- 
egories must be fed into ICC-mail by a "knowledge 

8 As of end of February 2000. 



engineer" , who maintains the system. The effects of 
new categories and new data have not been tested 

yet. 

The optimum performance of ICC-mail can be 
achieved only with a well-maintained category sys- 
tem. For a call center, this may be a difficult task 
to achieve, espescially under severe time pressure, 
but it will pay off. In particular, all new categories 
should be added, outdated ones should be removed, 
and redundant ones merged. Agents should only use 
these categories and no others. The organizational 
structure of the team should reflect this by defin- 
ing the tasks of the "knowledge engineer" and her 
interactions with the agents. 

6 Conclusions and Future Work 

We have presented new combinations of STP and 
SML methods to classify unrestricted e-mail text ac- 
cording to a changing set of categories. The current 
accuracy of the ICC-MAIL system is 78% (correct so- 
lution among the top five proposals), corresponding 
to an overall performance of 73% since ICC-mail 
processes only 94% of the incoming e-mails. The 
accuracy improves with usage, since each relearning 
step will yield better classifiers. The accuracy is ex- 
pected to approximate that of the agents, but not 
improve on it. With ICC-MAIL, the performance of 
an experienced agent can approximately be doubled. 

The system is currently undergoing extensive tests 
at the call center of AOL Bertelsmann Online. De- 
tails about the development of the performance de- 
pending on the throughput and change of categories 
are expected to be available by mid 2000. 

Technically, we expect improvements from the fol- 
lowing areas of future work. 

• Further task-specific heuristics aiming at gen- 
eral structural linguistic properties should be 
defined. This includes heuristics for the identi- 
fication of multiple requests in a single e-mail 
that could be based on key words and key 
phrases as well as on the analysis of the doc- 
ument structure. 

• Our initial ex periments with the integr ation 
of GermaNet ( Hamp and Feldweg, 1997 ), the 
evolving German version of WordNet, seem to 
confi rm the positive results described for W ord- 
Net ( de Buenaga Rodriguez et al., 1997 ) and 
will thus be extended. 

• A reorganization of the existing three-level cate- 
gory system into a semantically consistent tree 
structure would allow us to explore the non- 
terminal nodes of the tree for multi-layered 
SML. This places additional requirements on 
the knowledge engineering task and thus needs 
to be thoroughly investigated for pay-off. 



• Where system-generated answers are acceptable 
to customers, a straightforward extension of 
ICC-MAIL can provide this functionality. For 
the application in hand, this was not the case. 

The potential of the technology presented extends 
beyond call center applications. We intend to ex- 
plore its use within an information broking assis- 
tant in document classification. In a further indus- 
trial project with German Telekom, the ICC-MAIL 
technology will be extended to process multi-lingual 
press releases. The nature of these documents will 
allow us to explore the application of more sophis- 
ticated language technologies during linguistic pre- 
processing. 
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